Spread spectrum watermark for embedded signalling

ABSTRACT

A watermark is embedded into audio/video/image/multimedia data using spread spectrum methodology. The watermark is extracted from watermarked data without the use of an original or unwatermarked version of the data by using spatial or temporal local averaging of the frequency coefficients of the watermarked data.

FIELD OF THE INVENTION

The present invention relates to digital watermarking of data includingaudio, video, image and multimedia data. Specifically, the inventionrelates to the extraction of a watermark of embedded data fromwatermarked data without using an original or unwatermarked version ofthe data.

BACKGROUND OF THE INVENTION

The proliferation of digitized media such as video, image, audio andmultimedia is creating a need for a security system which facilitatesthe identification of the source of the material.

Context providers, i.e. owners of works in digital data form, have aneed to embed signals into audio/video/image/multimedia data which cansubsequently be recorded or detected by software and/or hardware devicesfor purposes of authenticating copyright ownership, control andmanagement.

For example, a coded signal might be inserted in data to indicate thatthe data should not be copied. The embedded signal should preserve theimage fidelity, be robust to common signal transformations and resistantto tampering. In addition, consideration must be given to the data ratethat can be provided by the system, though current requirements arerelatively low--a few bits per frame.

In U.S. patent application Ser. No. 08/534,894, filed Sep. 28, 1995,entitled "Secure Spread Spectrum Watermarking for Multimedia Data" nowabandoned and assigned to the same assignee as the present invention,there was proposed a spread spectrum watermarking method which embeddeda watermark signal into perceptually significant regions of an image forthe purposes of identifying the content owner and/or possessor. Astrength of this approach is that the watermark is very difficult toremove. In fact, this method only allows the watermark to be read if theoriginal image or data is available for comparison. This is because theoriginal spectrum of the watermark is shaped to that of the imagethrough a non-linear multiplicative procedure and this spectral shapingmust be removed prior to detection by matched filtering and thewatermark is inserted into the N largest spectral coefficients, theranking of which is not preserved after watermarking. Thus, this methoddoes not allow software and hardware devices to directly read embeddedsignals.

In an article by Cox et al., entitled "Secured Spectrum Watermarking forMultimedia", NEC Research Institute Inc., Technical Report 95-10, spreadspectrum watermarking is described which embed a pseudo-random noisesequence into the digital data for watermarking purposes.

The prior art watermark extraction methodology requires the originalimage spectrum be subtracted from the watermark image spectrum. Thisrestricts the use of the method when there is no original image ororiginal image spectrum available. One application where this presents asignificant difficulty is for third party device providers desiring toread embedded information for operation or denying operation of such adevice.

The present invention extends the earlier work of Cox et al to allow thereading or extraction of embedded signals by devices which do notcontain original data, e.g. original images.

In U.S. Pat. No. 5,319,735 by R. D. Preuss et al entitled "EmbeddedSignalling" digital information is encoded to produce a sequence of codesymbols. The sequence of code symbols is embedded in an audio signal bygenerating a corresponding code signal representing the sequence of codesymbols. The frequency components of the code signal being essentiallyconfined to a preselected signalling band lying within the bandwidth ofthe audio signal and successive segments of the code signal correspondsto successive code symbols in the sequence. The audio signal iscontinuously frequency analyzed over a frequency band encompassing thesignalling band and the code signal is dynamically filtered as afunction of the analysis to provide a modified code signal withfrequency component levels which are, at each time instant, essentiallya preselected proportion of the levels of the audio signal frequencycomponents in corresponding frequency ranges. The modified code signaland the audio signal are combined to provide a composite audio signal inwhich the digital information is embedded. This component audio signalis then recorded on a recording medium or is otherwise subjected to atransmission channel.

SUMMARY OF THE INVENTION

The present invention overcomes the limitations of the prior systems byusing spread spectrum technology to embed watermark data or informationinto predetermined locations in an image.

More specifically, the invention provides a system for extracting awatermark from watermarked data without using an original orunwatermarked version of the data.

The preferred method of watermarking extraction is to use a spatial ortemporal local average of the frequency coefficients of the watermarkeddata to determine the watermark. The frequency coefficients of atwo-dimensional neighborhood in two-dimensional watermarked data (e.g.an image), for example, are analyzed to reproduce the entire watermark.This is possible since the watermark is embedded into the data usingspread spectrum technology which places the watermark throughout thedata.

The invention is applicable to the watermarking ofaudio/video/image/multimedia data.

The invention will be best understood when the following description isread in conjunction with the accompanying drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of a method of inserting a watermarkinto an image;

FIG. 2 is a graphical representation of the image spectrum and shapedwatermark spectrum;

FIG. 3 is a schematic block diagram of a combiner;

FIG. 4 is a schematic block diagram of a method of extracting awatermark from a watermarked image;

FIG. 5 is a schematic block diagram of a separator;

FIG. 6 is a schematic block diagram of a spread spectrum system for usein watermark insertion;

FIG. 7 is a schematic block diagram of a spread spectrum receiver,

FIG. 8 is an original image to be watermarked;

FIG. 9 is the image in FIG. 8 after being watermarked; and

FIG. 10 shows a 4×4 array indicating the sequence of coefficients usedto form a one-dimensional vector.

DETAILED DESCRIPTION

Referring now to the figures and to FIG. 1 in particular, there is showna schematic block diagram of a method for inserting a watermark into adigital data, for instance an image. In the following descriptionreference may be made to image data or images. While the invention hasapplicability to image data and images, it will be understood that theteachings herein and the invention itself are equally applicable toaudio, video, image and multimedia data and the term image and imagedata will be understood to include these terms where applicable. As usedherein watermark will be understood to include embedded data, symbols,images, instructions or any other identifying information.

The image 10 is first transformed into a spatial frequencyrepresentation 12, for instance by a discrete cosine transform (DCT),other transforms such as a fast Fourier transform could also be used.The spectrum is then analyzed to determine the perceptually mostsignificant components 14 and the watermark to be imbedded 16 is thencombined 18 with the perceptionally most significant components. Thewatermark is a pseudo random number sequence (PNS) preferably chosenfrom a Gaussian distribution. After being combined, the modified imageis then inverse transformed back into the spatial domain to create thewatermarked image 20.

There are different ways to combine the watermark with the imagespectrum. In the preferred embodiment, the watermark components, W_(i),are added to the frequency coefficients, ƒ_(i), in a non-linear manneras

    ƒ.sub.i '=ƒ.sub.i +αƒ.sub.i W.sub.i(1)

where α is a constant typically in the range of 0.1 to 0.01. Inprinciple, α might also vary as a function of frequency and perceptualmodeling. Equation (1) can be considered a form of spectral shaping.That is, the original Gaussian white spectrum of the watermark is shapedto match that of the image by the second term in Equation (1) prior toaddition of the two spectra. The constant a serves as a gain control toadjust the relative strength of the two spectra. This is graphicallyshown in FIG. 2.

The two stages of the combiner are shown in FIG. 3. The watermark to beembedded into the data is provided as a first input to a spectral shaper30. The spectrum of the image to be watermarked is provided as a secondinput to the shaper 30. The output of shaper 30 is provided as a firstinput to summer 32. The spectrum of the image is provided as the secondinput to summer 32. The output of summer 32 is a watermarked spectrum.

To extract the watermark, the inverse process must be applied as shownin FIG. 4. The separator stage inverts the combiner stage. In order toextract the watermark components, W_(i), from a possibly distortedwatermark image, first subtract the original image before dividing bythe image spectral coefficients. The latter process serves to normalizeor equalize the watermark spectrum back to its original shape. That is

    W.sub.i =(ƒ.sub.i '-ƒ.sub.i)/αƒ.sub.i(2)

Specifically, the watermarked image 40 is transformed by a discretecosine transform or other transformation such at FFT, into a watermarkedimage spectrum 42. The stored original image 44 is transformed into anoriginal image spectrum 46.

The watermarked image spectrum 42 and the original image spectrum 46 areprovided as inputs to separator 50. The separator, as shown in FIG. 5,subtracts the original image spectrum from the watermarked imagespectrum 54 to obtain a difference image spectrum prior to normalizingthe resultant image. The spectral normalization 56 divides thedifference image spectrum by the image spectral coefficients αƒ_(i) toyield an extracted watermark.

The extracted watermark is statistically compared with the knowninserted watermark to calculate a statistical confidence level. Thestatistical confidence level provides a measure of whether the externalwatermark is the actual inserted watermark.

In the above described method of extracting a watermark, it is necessaryto have an original unwatermarked image. This is both an advantage andlimitation of the method. It is advantageous because it is difficult fora non-possessor of the image to remove the watermark. A limitation ofthe method is that it prevents a third party's software or hardwaredevices from extracting or reading the embedded signal information. Theuse of a Gaussian noise distribution is important for extraction usingan original image.

The above described prior art system is a special case of a more generalspread spectrum communication system in which the watermark informationis considered as the signal and the image is considered as the noise.FIG. 6 is a schematic block diagram of a special spectrum communicationsystem for use in watermark insertion.

In FIG. 6 a watermark signal is provided as an input to an errorcorrection encoder 60. The output of encoder 60 is provided to a spreadspectrum modulator 62. The output of modulator 62 is provided to aspectral transformation 64. The output of spectral transformation 64 isprovided as one input to a spectral shaper 66. A signal to bewatermarked is provided to a spectral transformer 68. The output of thetransformer 68 is provided as a second input to spectral shaper 66 andto a delay 70. The output of the spectral shaper 66 is added to theoutput of delay 70 at a summer 72. The summer output is subject to aninverse transform 74. The result of the inverse transform is awatermarked signal.

In the prior systems, the object was to embed a single PN (pseudo randomnumber) sequence into an image. The information associated with the PNsequence was assumed to be stored in a database together with theoriginal image and the spectral location of the embedded watermark. Thelocations of the watermarked components had to be recorded because theimplementation approximated the N perceptionally most significantregions of the watermark by the N largest coefficients. However, thisranking was not invariant to the watermarking process. The N largestcoefficients may be different after inserting the watermark than beforeintersecting the watermark.

In order to avoid this problem, the current method places a watermark inpredetermined locations of the spectrum, typically the first Ncoefficients, However, any predetermined locations could be used eventhough such locations should belong to the perceptually significantregions of the spectrum if the watermark is to survive common signalstransformations such as compression, scaling, etc.

More generally, the information to be embedded is a sequence of msymbols drawn from an alphabet A (e.g. the binary digits or the ASCIIsymbols). This data is then supplemented with additional symbols forerror detection and correction. Each symbol is then spread spectrummodulated, a process that maps each symbol into a unique PN sequenceknown as a chip. The number of bits per chip is preset--the longer thechip length, the higher the detected signal-to-noise ratio will be, butthis is at the expense of signaling bandwidth.

The spectrum of the PN sequence is white, i.e. flat, and is thereforeshaped to match that of the "noise", i.e. the image/video/audio/ormultimedia data into which the watermark is to be embedded. It is thisspectral shaping that must be modified from the prior methods so thatthe extraction process no longer requires the original image. To dothis, Equation (1) is modified so that each coefficient of thewatermarked spectrum is scaled by the local average of the imagespectral coefficient rather than the coefficient itself, i.e.

    ƒ.sub.i '=ƒ.sub.i +αavg(ƒ.sub.i)W.sub.i(3)

This average may be obtained in several ways. It may be a local averageover a two dimensional region. Alternatively, the two dimensionalspectrum may be sampled to form a one dimensional vector and a onedimensional local average may be performed. The latter method was usedin experimental results below. The average may be a simple box orweighted average over the neighborhood.

For video data, temporal averaging of the spectral coefficients overseveral frames can also be applied. However, since several frames areneeded for averaging at the spectral normalization stage of theextractor, the protection of individual video frames taken in isolationmay not be possible. For this reason, the present invention treats videoas a very large collection of still images. In this way, even individualvideo frames are copy protected.

Receiving or extracting a spread spectrum signal is shown in FIG. 7. Thewatermarked image, video, audio or multimedia data is first spectrallynormalized 76 to undo any previously performed spectral shaping. Thenormalized signal is then analyzed by a bank of correlators 78A . . .78Z, each correlator detecting the presence, if any, of a particular PNsequence (one for each symbol in the alphabet). The decision circuit 80typically selects the correlator with the maximum output as the mostlikely current symbol. More sophisticated decision procedures arepossible. The sequence of most likely current symbols is then providedas an input to an error correction stage 82 which corrects for falsedecisions made by the decision circuit. The output of the errorcorrection stage is an extracted watermark signal. In order to performthe spectral normalization 76, the previously performed spectral shapingprocedure is inverted. In the present case, the original unwatermarkedsignal is no longer available. Thus, an average of the frequencycoefficients, avg(ƒ_(i)), as approximated by the average of thewatermarked signal, i.e. avg(ƒ_(i))

    avg(ƒ.sub.i)˜avg(ƒ.sub.i ')        (4)

This is approximately true since the second term of equation (3) issmall relative to the first i.e.

    αavg(ƒ.sub.i)W.sub.i <<ƒ.sub.i     (5)

The normalization stage then divides each coefficient (ƒ_(i) ') in thereceived signal by the local average avg (ƒ_(i) ') in the neighborhood.

That is, ##EQU1##

The first term, on the right hand side (RHS) of Equation (6), ##EQU2##is considered a noise term. It was not present in the prior systemsbecause access to the unwatermarked coefficients allowed this term to beremoved. The second term αW_(i) is the original watermark signal whichcan now be detected using conventional correlation.

FIG. 8 shows an original image before being watermarked. FIG. 9 is thesame image after being watermarked in accordance with the teachings ofthe present invention.

The watermark in FIG. 9 was inserted using a gain of α=0.1 and a chiplength of 10,000. The first 10,000 coefficients of the original imagewere extracted in the sequence shown in FIG. 10 to form a onedimensional vector. A block average was then computed over a rectangularwindow of ±3 coefficients. The same procedure was applied at theextraction stage.

The correlator responds to randomly generated PN sequences with one suchsequence being set to the originally inserted sequence indicated. A verystrong and unambiguous response on 0.125 was detected for a particularPN sequence. For uncorrelated watermarks, the correlator output isapproximately Normally distributed with a variance of 1/(N-2), where Nis the length of the watermark. Thus, for N=10,000, the standarddeviation is 0.01 and the correlation response of 0.125 represents over12 standard deviations. A response of approximately 30 deviations wasachieved with the prior art method using a watermark length of only1,000. The reduction in signal-to-noise ratio is due almost entirely tothe first term of the right-hand side of Equation (6) which is notpresent when using the prior art method.

In order to determine the closeness of the approximation of Equation(4), the equalization process was repeated using the original shapingcoefficients, i.e. avg(ƒ_(i)), instead of avg (ƒ_(i) '). The correlatorresponse increase from 0.125 to 0.15, suggesting that a loss ofapproximately 20% was incurred due to this approximation. Of course,this loss is strongly dependent of the local smoothness in the spectraof an image and may vary significantly from image to image.

In summary, the present invention provides a modification to existingdigital watermarking methods in which the original data was required forwatermark extraction thereby enabling watermarking extraction in theabsence of an unwatermarked or original data. The present invention useslocal spatial and/or temporal local averaging of the frequencycoefficients. The result is extraction of the watermark with very highconfidence.

While there has been described and illustrated a system for inserting awatermark into and extracting a watermark from watermarked data withoutusing an unwatermarked version of the data, it will be apparent to thoseskilled in the art that variations and modifications are possiblewithout deviating from the broad principles and teachings of the presentinvention which shall be limited solely by the scope of the claimsappended hereto.

What is claimed is:
 1. A method of extracting a watermark fromwatermarked data comprising the steps of:receiving watermarked data;spectrum normalizing the watermarked data to generate a normalizedsignal; correlating the normalized signal with predetermined PNsequences corresponding to predetermined symbols to provide correlatedsignals for each predetermined PN sequence; deciding which PN sequenceis present in the watermarked data based upon said correlated signals;and extracting a watermark corresponding to a sequence of PN sequencespresent.
 2. A method of extracting a watermark from watermarked data asset forth in claim 1, where said spectrum normalizing removes anyspectral shaping performed when the watermark was inserted into thewatermarked data.
 3. A method of extracting a watermark from watermarkeddata as set forth in claim 1, where said spectrum normalizing determineslocal average of the frequency coefficients of the spectrum of thewatermarked data and then divides each frequency coefficient of thespectrum of the watermarked data by the local average of the frequencycoefficients.
 4. A method of extracting a watermark from watermarkeddata as set forth in claim 1, where said watermarked data is audio data.5. A method of extracting a watermark from watermarked data as set forthin claim 1, where said watermarked data is video data.
 6. A method ofextracting a watermark from watermarked data as set forth in claim 1,where said watermarked data is image data.
 7. A method of extracting awatermark from watermarked data as set forth in claim 1, where saidwatermarked data is multimedia data.
 8. A method for inserting awatermark into data comprising the steps of:providing a watermark;providing a spectrum of the data; shaping the spectrum of the watermarkcommensurate with the average spectrum of the data; and combining theshaped spectrum watermark with the spectrum of the data to obtainwatermarked data spectrum.
 9. A method for inserting a watermark intodata as set forth in claim 8, where said average is a local average overa two dimensional region of said data.
 10. A method for inserting awatermark into data as set forth in claim 9, where said average is aweighted average over a neighborhood of said data.
 11. A method forinserting a watermark into data as set forth in claim 8, where saidaverage is a temporal average over several frames of video data.
 12. Amethod for inserting a watermark into data as set forth in claim 8,where said data is audio data.
 13. A method for inserting a watermarkinto data as set forth in claim 8, where said data is video data.
 14. Amethod for inserting a watermark into data as set forth in claim 8,where said data is image data.
 15. A method for inserting a watermarkinto data as set forth in claim 8, where said data is multimedia data.16. A method of inserting a watermark into data comprising the stepsof:providing a watermark signal, providing a spectrum of the data;shaping the spectrum of the watermark signal to be commensurate with thespectrum of the data; and summing the shaped spectrum watermark signalwith the spectrum of the data to obtain watermarked data spectrum.
 17. Amethod of inserting a watermark into data as set forth in claim 16,where said shaping the spectrum of the watermark shapes the spectrumcommensurate with temporal averages of frequency coefficients of thespectrum of the data.
 18. A method of inserting a watermark into data asset forth in claim 16, where said shaping the spectrum of the watermarksignal shapes the spectrum commensurate with local averages of frequencycoefficients of the spectrum of the data.
 19. A method of inserting awatermark into data as set forth in claim 16, where said data is audiodata.
 20. A method of inserting a watermark into data as set forth inclaim 16, where said data is video data.
 21. A method of inserting awatermark into data as set forth in claim 16, where said data is imagedata.
 22. A method of inserting a watermark into data as set forth inclaim 16, where said data is multimedia data.